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Abstract1 

As  the  systems  engineering  community  continues  to  mature  model-based  approaches, 
exciting  opportunities  for  sophisticated,  computational  analysis  grow.  Among  these 
possibilities,  we  posit  and  demonstrate  a  novel  algorithm  for  estimating  the  cost  of 
architectural  changes  early  in  the  system  life  cycle  when  uncertainty  is  high.  In  particular,  by 
treating  the  DoD  Architecture  Framework  (DoDAF)  Systems  View  3  (SV3)  as  an  adjacency 
matrix,  we  leverage  concepts  from  network  science  to  analyze  the  impact  of  architectural 
changes  that  result  from  the  addition  of  a  subsystem.  Following  this  growth,  we  estimate  the 
marginal  increase  in  systems  engineering  effort  via  an  explicit  connection  between  the  open 
academic  cost  model  COSYSMO  (Constructive  Systems  Engineering  Cost  Model)  and  the 
SV3.  Based  on  its  stochastic  nature,  this  procedure  is  further  implemented  as  a  Monte  Carlo 
simulation,  allowing  us  to  generate  distributions  of  potential  cost  growth.  Theoretically,  this 
work  serves  as  a  proof  of  concept  for  further  research  on  the  integration  of  network  science 
and  systems  engineering.  Practically,  the  methodology  provides  a  means  for  practitioners  to 
accelerate  the  accuracy  and  fidelity  of  their  “should  cost”  and  “will  cost”  analyses  early  in  the 
systems  life  cycle. 

Introduction 

On  August  2,  201 1 ,  President  Barack  Obama  signed  the  Budget  Control  Act  (201 1 ) 
into  law.  Driven  by  a  need  to  reign  in  federal  spending  and  curtail  the  growth  of  the  national 
debt,  the  law  contained  a  provision  for  a  decade’s  worth  of  automatic,  wide-sweeping,  and 
substantial  budget  cuts,  if  Congress  failed  to  pass  a  deficit  reduction  bill  (Heniff,  Rybicki,  & 


1This  work  is  derived  from  papers  presented  at  the  Conference  of  Systems  Engineering  Research  (CSER)  over 
the  past  two  years  (Dabkowski,  M.,  Estrada,  J.,  Reidy,  B.,  &  Valerdi,  R.,  2013;  Dabkowski,  M.,  Valerdi,  R.,  & 
Farr,  J.,  2014).  In  particular,  material  in  the  sections  titled  “COSYSMO — A  Tool  for  Costing  Architectural 
Complexity,”  “Network  Science — A  Mechanism  for  Generating  Unforeseen  Architectural  Growth,”  and 
“Estimating  the  Cost  of  Architectural  Growth”  originally  appeared  in  Elsevier’s  Procedia  Computer  Science  (with 
copyright  retained  by  the  authors).  With  this  in  mind,  the  first  three  sections  of  this  paper  add  substantial  political 
context  and  acquisition  background,  while  the  last  few  sections  cover  new  limitations  and  possibilities  for  future 
work. 
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Mahan,  2011,  pp.  2-3).  Dubbed  sequestration,  the  provision  was  intended  as  a  forcing 
function  to  generate  congressional  consensus  (Heniff  et  al.,  201 1 ,  p.  27);  it  failed.  Thus,  on 
March  1, 2013,  President  Obama  implemented  the  cuts  (Sequestration,  2013),  and  the 
Department  of  Defense  (DoD)  absorbed  an  immediate,  unanticipated  23%  reduction  in  its 
Fiscal  Year  2013  budget  (Office  of  the  Secretary  of  Defense,  2011),  as  well  as  a  combined 
$492  billion  loss  in  funding  over  the  next  10  years  (Heniff  et  al.,  2011,  p.  30). 

Of  course,  the  impact  of  this  loss  on  defense  acquisition  is  substantial,  as  the 
combined  services  and  the  industrial  base  face  furloughs,  reduced  production,  and  difficult 
modernization  decisions  (On  Impacts,  2013).  That  said,  according  to  Dr.  William  LaPlante 
(Principal  Deputy  Assistant  Secretary  of  the  Air  Force  [Acquisition])  and  Lieutenant  General 
(LTG)  Michael  Moeller  (United  States  Air  Force  Deputy  Chief  of  Staff,  Strategic  Plans  and 
Programs),  the  “single  largest  impact  of  sequestration  and  current  budgetary  unknowns  is 
[their  effect  on] .  .  .  the  meticulous  cost  and  schedule  planning  mandated  in  numerous  public 
laws  and  DoD  acquisition  policy  directives”  (On  Impacts,  2013,  p.  70). 

To  be  sure,  Dr.  LaPlante  and  LTG  Moellers’  assertion  has  tremendous,  historical 
support.  After  all,  even  in  the  best  of  times,  cost  estimation  (and  its  correlate  scheduling)  is 
difficult.  For  example,  consider  that  between  1997  and  2009,  47  major  defense  acquisition 
programs  (MDAPs)  experienced  cost  overruns  of  at  least  15%  or  30%  over  their  current  or 
original  baseline  estimates,  respectively  (GAO,  201 1 ,  p.  2).  Known  formally  as  a  Nunn- 
McCurdy  breach,  the  reasons  for  this  excessive  growth  are  myriad,  although  nearly  70%  of 
the  cases  identified  engineering  and  design  issues  as  a  contributing  factor  (GAO,  2011,  p. 

5). 

In  sum,  our  defense  budget  is  uncertain  and  presumably  shrinking;  uncertain  funding 
frustrates  cost  planning;  and  cost  planning  (however  meticulous)  is  already  difficult  and 
often  based  on  incomplete  information.  Unfortunately,  in  an  uncertain  environment,  the  need 
to  plan  well  is  paramount.  Indeed,  we  find  ourselves  in  challenging  times. 

Pre-Milestone  A  Cost  Estimation — Filled  With  Potential  and  Fraught  With  Peril 

While  the  fiscal  emergencies  of  the  past  several  years  have  placed  a  spotlight  on 
defense  acquisition  and  spending  in  general,  Congress  legislatively  acknowledged  the  need 
for  change  in  2009  with  the  passage  of  the  Weapon  Systems  Acquisition  Reform  Act 
(WSARA,  2009).  Generally  speaking,  the  WSARA  aims  to  reduce  cost  overruns  and 
schedule  delays  through  a  collection  of  four  major  organizational  changes  and  seven 
procedural  adjustments  (Berteau,  Hofbauer,  &  Sanok,  2010,  p.  4).  For  the  purpose  of  this 
section,  several  stand  out. 

First,  the  WSARA  increased  the  rigor  and  accountability  of  Pre-Milestone  A  (Pre-MS 
A)  analysis  and  certification.  For  example,  as  directed  by  Ashton  Carter,  the  former  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]),  “the 
[Milestone  Decision  Authority]  MDA  for  an  MDAP  shall  sign  a  memorandum  with  the  subject 
‘Milestone  A  Program  Certification,’  prior  to  signing  the  [acquisition  decision  memorandum] 
ADM  to  approve  MS  A”  (Under  Secretary  of  Defense  [AT&L],  2009,  p.  11).  Additionally,  as 
part  of  this  process,  the  MDA  must  include  the  following  language  in  the  ADM:  “At  any  time 
prior  to  MS  B  approval,  the  [program  manager]  PM  shall  notify  me  immediately  if  the 
projected  cost  of  the  program  exceeds  the  cost  estimate  for  the  program  at  the  time  of  MS  A 
certification  [emphasis  added]  by  at  least  25  percent”  (Under  Secretary  of  Defense  [AT&L], 
2009,  p.  11).  Moreover,  if  such  growth  and  notification  occurs,  then  the  MDA  must  notify 
Congress  and  either  (a)  defend  the  program  and  recommend  its  continuation  or  (b)  suggest 
a  plan  for  termination  (Under  Secretary  of  Defense  [AT&L],  2009,  p.  12).  Simply  put,  as  per 
the  WSARA,  Pre-MS  A  cost  estimation  is  a  critical  go/no  go  decision  point. 
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Next,  the  WSARA  established  a  new  position  known  as  the  Performance 
Assessments  and  Root  Cause  Analysis  (PARCA)  official. 2  Functionally,  the  PARCA 
conducts  reviews  of  an  MDAP’s  cost,  schedule,  and  performance,  as  well  as  root  cause 
analysis  “including  the  role  of .. .  excessive  manufacturing  or  integration  risk  [and] 
unanticipated  design,  engineering,  manufacturing,  or  integration  issues  [emphasis  added]” 
(Under  Secretary  of  Defense  [AT&L],  2009,  pp.  7-8).  Moreover,  the  WASRA  required  the 
Director  of  Defense  Research  and  Engineering  (DDR&E)  to  develop  “standards  that  will  be 
used  to  measure  and  assess  the  maturity  of  critical  technologies  and  integration  risk 
[emphasis  added]  in  MDAPs”  (Under  Secretary  of  Defense  [AT&L],  2009,  p.  9).  In  short,  the 
WSARA  is  quite  clear — assessing  integration  and  design  risk  matters! 

With  the  exceptions  of  a  PARCA  root  cause  analysis  following  critical  cost  growth 
and  a  DDR&E  technological  maturity  evaluation  prior  to  MS  B,  the  timing  of  the  PARCA  and 
DDR&Es’  assessments  are  somewhat  vague,  being  described  as  “periodic,”  “at  key  stages,” 
or  “upon  request”  (Under  Secretary  of  Defense  [AT&L],  2009,  pp.  7-9).  That  said,  given  (a) 
the  increased  emphasis  on  Pre-MS  A  cost  estimation  and  integration  risk  and  (b)  the 
recurring  role  of  engineering  and  design  issues  in  excessive  cost  growth,  it  seems 
reasonable  to  require  an  assessment  of  integration  and  design  risk  Pre-MS  A.  This 
conclusion  is  implicitly  reinforced  by  additional  WSARA  guidance  mandating  that  Analysis  of 
Alternatives  (AoA)  give  “[f]ull  consideration  of  possible  trade-offs  among  cost,  schedule,  and 
performance  objectives  for  each  alternative  considered”  (Under  Secretary  of  Defense 
[AT&L],  2009,  p.  3;  Under  Secretary  of  Defense  [AT&L],  2013b,  p.  122). 


Figure  1.  Illustration  of  the  Interaction  Between  the  Capability  Requirements  Process 

and  the  Acquisition  Process 

(Under  Secretary  of  Defense  [AT&L],  2013b,  p.  5) 


2  The  WSARA  also  established  a  Director  of  Cost  Assessment  and  Program  Evaluation  (DCAPE),  Director  of 
Development  Test  and  Evaluation  (DT&E),  and  a  Director  of  Systems  Engineering  (SE)  (WSARA,  2009); 
however,  these  positions  are  less  relevant  to  this  section. 
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Conceptually,  conducting  an  assessment  of  integration  and  design  risk  Pre-MS  A  is 
appealing.  After  all,  as  shown  in  Figure  1,  Pre-MS  A  is  commensurate  with  the  Material 
Solution  Analysis  Phase,  during  which  “affordability  analysis,  risk  analysis,  and  planning  for 
risk  mitigation  are  key  activities”  (Under  Secretary  of  Defense  [AT&L],  2013b,  p.  15). 
Moreover,  during  a  system’s  conceptual  or  preliminary  design  phase,  the  majority  of  a 
program’s  future,  life-cycle  costs  are  often  (and  perhaps  unwittingly)  committed  (Blanchard 
&  Fabrycky,  1998,  p.  37;  Dowlatshahi,  1992,  p.  1803).  Put  another  way,  today’s  design 
decisions,  driven  by  current  requirements,  determine  tomorrow’s  debts,  and  the  implication 
is  obvious — make  better  design  decisions  now. 

Unfortunately,  this  is  easier  said  than  done.  Specifically,  while  the  early  life  cycle 
contains  substantial  opportunities  for  future  cost  savings,  it  is  characterized  by  uncertainty, 
notably  in  what  will  ultimately  be  required.  For  example,  based  on  workshops  conducted 
between  2010  and  201 1  involving  “27  participants  .  .  .  [with]  an  average  of  23  years  of 
experience  in  variety  of  industries  but  with  specific  emphasis  on  aerospace  and  defense,”  an 
average  of  28%  of  a  system’s  baseline  requirements  will  change  over  the  course  of  its  life 
cycle,  with  roughly  43%  of  these  changes  occurring  in  the  development  phase  (Pena  & 
Valerdi,  2014,  pp.  16-18).  In  the  parlance  of  the  Defense  Acquisition  System  (DAS)  and  as 
seen  in  Figure  1 ,  these  changes  occur  Post-MS  B.  Furthermore,  when  system  requirements 
change,  this  change  manifests  itself  as  the  modification  of  an  existing  requirement  or  the 
addition  of  a  new  requirement  86%  of  the  time  (Pena  &  Valerdi,  2014,  p.  17). 

Quite  bluntly,  experience  suggests  we  are  much  more  likely  to  add  than  take  away 
(i.e.,  scope  creep),  and  these  changes  often  carry  substantial  costs.  For  instance,  in  their 
meticulous  RAND  study,  Bolten,  Leonard,  Arena,  Younossi,  and  Sollinger  carefully 
examined  the  Selected  Acquisition  Reports  (SARs)  of  35  MDAPs,  attributing  differences 
between  their  current  and  MS  B  cost  estimates  to  one  of  13  categories  (2008,  pp.  14-15). 
Among  these  categories  is  requirements  volatility  (dubbed  “requirements”),  which  captures 
requirement  changes  that  occur  “at  any  point  after  MS  B  . .  .  [and  normally]  add  capabilities 
to  the  system”  (Bolten  et  al.,  2008,  p.  17).  For  the  35  MDAPs  analyzed,  requirements 
volatility  accounted  for  an  average  of  21 .5%  of  a  program’s  total  cost  growth  over  its  MS  B 
estimate,  making  it  the  second  largest  source  of  cost  growth  (RAND,  2008,  p.  27). 
Occasionally,  as  evidenced  by  the  C130J  “Super  Hercules,”  a  change  in  requirements  can 
necessitate  the  addition  of  a  subsystem,  and  the  resulting  cost  growth  can  be  extreme. 

The  program  originally  was  envisioned  as  a  nondevelopmental  aircraft 
acquisition  with  a  negligible  [Research,  Development,  Test,  and  Evaluation] 
RDT&E  effort  planned.  Several  years  into  the  program,  the  decision  was 
made  to  install  the  Global  Air  Traffic  Management  system,  adding  several 
hundred  million  dollars  to  development  and  causing  the  total  development 
cost  growth  to  climb  upward  of  2,000  percent.  (Under  Secretary  of  Defense 
[AT&L],  2013a,  p.  25) 

With  this  in  mind,  it  is  no  surprise  that  Pre-MS  A  cost  estimation  is  challenging,  and 
its  complications  have  been  duly  noted  by  analysts  in  the  Office  of  the  Deputy  Assistant 
Secretary  of  the  Army  for  Cost  and  Economics  (ODASA-CE;  Hull,  2009;  Roper,  2010)  as 
well  as  Carnegie  Mellon’s  Software  Engineering  Institute  (SEI)  (Ferguson,  Goldenson, 
McCurley,  Stoddard,  Zubrow,  &  Anderson,  2011,  pp.  6-7).  Collectively,  these  issues  stem 
from  a  lack  of  system  specification  prior  to  MS  A,  making  traditional,  bottom-up  estimation 
difficult  at  best.  As  such,  ODASA-CE  developed  a  parametric  method  known  as  capability- 
based  cost  analysis,  where  the  cost  to  acquire  a  desired  set  of  capabilities  is  estimated  from 
the  historical  cost  of  acquiring  them  (Hull,  2009;  Roper,  2010).  Taking  a  multimethodology 
approach,  SEI  developed  QUELCE  (Quantifying  Uncertainty  in  Early  Lifecycle  Cost 
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Estimation),  which  incorporates  expert  opinion,  Bayesian  belief  networks,  parametric  cost 
models,  and  Monte  Carlo  simulation  (Ferguson  etal.,  2011). 

While  these  methods  are  tremendously  valuable  and  should  continue  to  be  refined, 
the  DoD’s  current  push  to  require  more  system  specification  earlier  in  the  lifecycle  has 
created  an  opportunity  for  an  alternative  approach.  In  particular,  while  the  USD(AT&L) 
previously  specified  that  the  initial  submission  of  an  MDAP’s  Capability  Development 
Document  (CDD)  was  prior  to  MS  B  (Under  Secretary  of  Defense  [AT&L],  2008,  p.  19),  the 
recently  signed  interim  DoD  Instruction  5000.02  now  requires  a  “draft”  or  DoD  component- 
approved  CDD  prior  to  MS  A  (see  Figure  1 ;  Under  Secretary  of  Defense  [AT&L],  2013b,  p. 
46).  Moreover,  according  to  the  current  Manual  for  the  Operation  of  the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS),  CDDs  must  contain  25  of  the  31  DoD 
Architecture  Framework  (DoDAF)  models  ever  required  by  JCIDS  (CJCS,  2012a,  p.  B-F-6), 
and  these  models  are  the  same  set  required  by  the  Capability  Production  Document  (CPD), 
which  is  submitted  just  prior  to  an  MDAP  entering  production  (see  Figure  1;  CJCS,  2012a,  p. 
B-F-6).3  Of  course,  the  Pre-MS  A,  draft  CDD  can  and  likely  will  change,  but  the  conclusion  is 
clear — more  information  is  available  earlier,  most  notably  detailed  architectural  models. 

Systems  Engineering — A  Discipline  With  Promise 

Given  the  bevy  of  DoDAF  models  now  available  Pre-MS  A,  the  natural  question  is 
“What,  if  anything,  can  these  draft  diagrams  tell  us  about  cost?”  While  an  exhaustive  answer 
to  this  query  requires  a  thorough  examination  of  each  of  the  25  models,  our  intent  is  to 
provide  a  proof  of  concept.  Accordingly,  we  focus  on  one — the  Systems  Viewpoint  2  (SV2): 
Systems  Resource  Flow  Description. 

As  a  matter  of  definition,  the  SV2  documents  “details  of  the  physical  pathways  or 
network  patterns  that  implement  interfaces”  (Department  of  Defense  Deputy  Chief 
Information  Officer,  2010,  p.  205).  Graphically,  this  normally  takes  the  form  of  a  block 
diagram,  and  an  example  is  given  in  Figure  2,  which  documents  the  flow  of  resources 
between  devices  in  the  Ocean  Observatories  Initiative’s  (OOI)  Portland  site.4  As  this 
example  shows,  the  SV2  visually  captures  how  the  subsystems  of  a  larger  system  (or 
program)  connect. 

That  said,  as  a  system’s  complexity  or  size  increases,  the  SV2  can  quickly  become 
unwieldy  and  unreadable,  forcing  the  architect  to  generate  a  library  of  smaller  SV2s.  In  such 
situations,  the  larger  number  of  program-wide  interfaces  can  be  more  compactly 
represented  in  the  Systems  Viewpoint  3  (SV3):  Systems-Systems  Matrix.  The  SV3  for  the 
OOI  Portland  site  is  given  in  Figure  3,  where  shaded  cells  indicate  that  the  subsystems  in 
the  corresponding  rows  and  columns  are  connected. 


3  Although  the  current  Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System 
(dated  January  19,  2012)  does  not  require  a  “DIV-3:  Physical  Data  Model”  for  the  CDD  but  does  require  it  for  the 
CPD,  this  is  an  error  (CJCS,  2012b,  p.  3).  It  is  required  for  both. 

4  The  OOI  is  a  National  Science  Foundation  project  that  will  ultimately  provide  a  “networked  infrastructure  of 
science-driven  sensor  systems  to  measure  the  physical,  chemical,  geological  and  biological  variables  in  the 
ocean  and  seafloor”  (Ocean  Observatories  Initiative,  2013). 
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Figure  2.  SV2  Documenting  the  Flow  of  Resources  for  OOl’s  Portland  Site 

(Farcas,  2011) 


Figure  3.  SV3  Summarizing  the  OOI  Portland  Site’s  SV2 
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Professionally  speaking,  the  SV2  and  SV3  fall  within  the  purview  of  systems 
engineering  (SE),  a  discipline  repeatedly  identified  as  a  means  to  control  program  cost.  For 
example,  in  the  Government  Accountability  Office’s  (GAO)  2011  report  DoD  Cost  Overruns, 
“early  and  continued  systems  engineering  analysis”  is  the  first  of  five  tools  recommended  for 
containing  cost  growth  (p.  7).  Similarly,  in  a  2008  National  Research  Council  (NRC)  report, 
the  authors  state  that  the  “application  of  SE  to  decisions  made  in  the  pre-Milestone  A  period 
is  critical  to  avoiding  (or  at  least  minimizing)  cost  and  schedule  overruns”  (p.  3).  These 
assertions  are  empirically  backed-up  by  data  from  historical,  software-intensive  systems, 
where  the  return  on  investment  for  greater  SE  effort  can  be  as  high  as  8:1  (Boehm,  B., 
Valerdi,  R.,  &  Honour,  E.,  2008,  p.  232).  Legislatively,  Congress  acknowledged  the  value  of 
SE  in  2009  by  adding  a  Director  of  Systems  Engineering  to  the  USD(AT&L)’s  staff  (WSARA, 
2009,  p.  1710). 

With  SE’s  cost  saving  potential  firmly  established,  the  aforementioned  NRC  report 
identifies  six  primary  “seeds  of  [cost,  schedule,  and  performance]  failure”  addressable  by 
SE,  including  incomplete  requirements  at  MS  B,  system  complexity  (via  internal, 
architectural  design),  and  external  interface  complexity  (via  network-centric  operations  or 
“systems  of  systems”  constructs;  2008,  pp.  82-85).  Abstractly,  the  SV3  provides  a  compact 
representation  of  all  three,  as  requirements  (however  incomplete)  drive  the  selection  of 
subsystems  and,  thus,  spawn  their  attendant  interfaces,  both  internal  and  external. 
Therefore,  formally  evaluating  the  interfaces  portrayed  in  the  SV3  and  subsequently 
estimating  their  cost  holds  promise.  Accordingly,  we  turn  our  attention  to  the  Constructive 
Systems  Engineering  Cost  Model  (COSYSMO). 

COSYSMO — A  Tool  for  Costing  Architectural  Complexity 

Developed  by  the  second  author  in  2005,  COSYSMO  is  a  parametric,  open 
academic  cost  model  that  estimates  the  systems  engineering  effort  (in  person  months) 
required  to  bring  a  system  to  fruition  (Valerdi,  2005).  Consisting  of  four  size  drivers  (i.e., 
number  of  requirements,  number  of  interfaces,  number  of  algorithms,  and  number  of 
operational  scenarios)  and  14  effort  multipliers,  it  has  been  used  by  a  variety  of 
organizations  (Valerdi,  2008;  Wang,  Valerdi,  Roedler,  Ankrum,  &  Gaffney,  2012),  and  it 
utilizes  the  cost  estimating  relationship  (CER)  given  in  Equation  1  below  (Valerdi,  2005): 


(1) 


where 


PMns  =  system  engineering  effort  (nominal  schedule), 

A  =  calibration  constant  derived  from  historical  project  data  (assume  as  0.25), 

Wik  =  weight  for  the  ith  complexity  level  of  the  /cth  size  driver  (i  e 
(e  (easy),  n  (nominal),  d  (difficult)}), 

<Pik  =  quantity  of  the  fcth  size  driver  with  complexity  level  i  ( k  e 
(1  (requirements),  2  (interfaces),  3  (algorithms)  and  4  (operational 
scenarios)}), 

E  =  diseconomies  of  scale  constant  (assume  as  1 .06),  and 

EMj  =  systems  engineering  effort  multiplier  for  the  ;'th  cost  driver  (assume 

product  is  0.89). 
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With  respect  to  costing  architectural  complexity  via  the  SV3,  we  are  primarily 
concerned  with  the  “number  of  interfaces”  size  driver,  where  system  interfaces  are  defined 
as  “shared  major  physical  and  logical  boundaries  between  system  components  or  functions 
(internal  interfaces)  and  those  external  to  the  system  (external  interfaces)”  (Valerdi,  2005,  p. 
282).  Additionally,  from  a  complexity  (and  thus  effort)  standpoint,  all  interfaces  are  not 
created  equal;  therefore,  we  categorize,  count,  and  weigh  them  according  to  the  taxonomy 
given  in  Table  I  (Valerdi,  2014,  p.  284). 


Table  1.  COSYSMO’s  Interface  Rating  Scale  and  Relative  Weights5 


Complexity  Level 

Characteristics 

Easy 

Nominal 

Difficult 

Message  complexity 

Simple 

Moderate 

Complex 

Coupling  level 

Uncoupled 

Loose 

Tight 

Stakeholder  consensus 

Strong 

Moderate 

Low 

Behavior 

Well  behaved 

Predictable 

Emergent 

Relative  weight  (w*) 

1.1 

2.8 

6.3 

Armed  with  COSYSMO’s  CER  and  its  corresponding  interface  rating  scale,  consider 
a  hypothetical  system  with  the  SV3  portrayed  Figure  4. 


Figure  4.  SV3  for  a  Hypothetical  System  Where  the  Shading  of  the  Cells  Indicates 
Interface  Complexity  (Light  Gray  =>  Easy,  Medium  Gray  =>  Nominal,  Black  => 

Difficult) 

Consisting  of  20  subsystems  (labeled  A  through  T)  and  47  interfaces,  assume  the 
system  is  mature  enough  for  interface  complexity  to  be  assessed.  Moreover,  without  loss  of 
generality,  assume  there  are  200  easy,  200  nominal,  and  50  difficult  requirements,  as  well 
as  5  difficult  critical  algorithms.  Using  additional  wik  and  EMj  data  obtained  from  Valerdi 
(2014,  p.  284),  we  apply  Equation  1  to  obtain  an  initial  estimate  of  PMNS  as  follows: 


5  The  relative  weights  in  Table  1  have  been  obtained  through  industry  calibration. 
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irquircmndi  algprilh  mi  intnficn 


4 


-0.89  =  245.27 


(2) 


As  such,  we  conclude  that  245.27  PM  of  SE  effort  are  required,  and,  if  we  further 
assume  that  each  PM  costs  $20,000,  this  equates  to  $4,905,400.  In  a  perfect  world,  our 
Pre-MS  A  requirements  would  be  well-defined,  our  architecture  would  hold,  and  this 
estimate  would  suffice.  Reality,  however,  is  rarely  this  kind,  as  our  earlier  discussion  on 
requirements  volatility  attests.  Nonetheless,  if  revised  requirements  prompt  the  elimination 
or  introduction  of  new  subsystems  and  interfaces,  the  resultant  cost  impact  can  be  easily 
quantified  since  there  is  an  explicit  connection  between  the  SV3  and  COSYSMO. 

Network  Science — A  Mechanism  for  Generating  Unforeseen  Architectural 
Growth 

While  the  above  methodology  is  useful,  it  fails  to  provide  decision  makers  with  a 
sense  of  how  costly  architectural  change  might  be  before  it  occurs.  For  example,  suppose 
we  are  interested  in  estimating  the  SE  effort  required  to  incorporate  an  additional  subsystem 
(U)  into  the  architecture  without  knowing  its  purpose  or  function.  In  light  of  COSYSMO’s 
CER,  this  ultimately  forces  us  to  estimate  the  number  of  interfaces  (by  complexity  level)  U 
will  generate.  More  granularly,  we  need  to  answer  the  following  three  growth  modeling 
questions: 

a.  How  many  subsystems  should  U  connect  to  (degree)?, 

b.  Given  U  connects  to  d  subsystems,  which  d  subsystems  should  it 
connect  to  (adjacency)?,  and 

c.  Given  U  connects  to  a  specific  set  of  d  subsystems,  what  should  the 
complexity  of  these  interfaces  be  (weights)? 

Of  course,  the  answer  to  these  questions  cannot  be  known  with  certainty,  as 
subsystem  U’s  purpose  is  unknown.  To  illustrate  this  visually,  consider  Figure  5,  which 
represents  six  distinct  possibilities  for  subsystem  U’s  connections. 


Figure  5.  Potential  Realizations  of  Subsystem  U’s  Connections 
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As  seen  in  Figure  5,  subsystem  U  can  connect  to  the  existing  architecture  in  a 
variety  of  ways.  For  example,  in  Panel  3  of  Figure  5,  U  connects  to  B,  L,  M,  and  Q  with 
complexities  easy,  nominal,  nominal,  and  difficult  respectively.  On  the  other  hand,  in  Panel 
6,  it  connects  to  nine  subsystems,  where  its  connection  to  I  is  rated  as  easy. 

This  raises  the  question:  Which  of  these  realizations  is  more  likely?  Of  course,  with 
no  information  on  the  function  or  purpose  of  U,  we  could  claim  neither.  However,  if  we  make 
the  reasonable  assumption  that  the  network’s  structure  prior  to  U’s  addition  provides  a 
reasonable  facsimile  of  its  structure  following  U’s  addition,  then  Panel  3  seems  to  be  the 
obvious  choice.  After  all,  while  40%  of  the  existing  subsystems  have  four  or  fewer  interfaces, 
none  have  more  than  eight.  Additionally,  although  each  of  I  s  existing  interfaces  are  rated  as 
nominal,  its  interface  to  U  in  Panel  6  is  categorized  as  easy. 

Beyond  the  obvious  inconsistencies  in  Panel  6  of  Figure  5,  the  manner  in  which 
systems  engineers  architect  complex  systems  should  also  be  taken  into  account.  As  the 
authors  of  the  2008  NRC  report  note  with  respect  to  system  architecture  development, 

[a]  well-known  approach  to  addressing  a  complex  problem  is  to  break  it  into 
parts  that  can  be  addressed  separately.  Architecture  here  refers  to  the 
partitioning  of  the  system  into  separately  definable  and  procurable  parts,  the 
structuring  of  interfaces  between  the  system  and  the  outside  world,  and  the 
structuring  of  interfaces  (physical,  functional,  and  data)  among  the  segments. 
Through  careful  partitioning,  architecture  can  minimize  complexity  and 
thereby  development  risk.  (p.  78) 

While  the  deliberate  or  subconscious  partitioning  of  the  hypothetical  system  is  not 
apparent  in  Figure  4,  we  can  use  the  well-known  Girvan-Newman  algorithm  from  network 
science  to  identify  the  system’s  “architectural  communities” — groups  of  subsystems  where 
the  number  of  interfaces  is  dense  within  and  sparse  between  groups.  Running  this 
procedure  on  our  hypothetical  system  identifies  three  communities,  namely:  community  1  = 
{Q,  O,  E,  L,  M,  J},  community  2  =  {N,  G,  H,  D,  I,  K,  T,  B,  S},  and  community  3  =  {F,  P,  A,  R, 
C}.6  Furthermore,  when  the  subsystems  are  permuted  by  their  community  membership 
(Figure  6),  the  partition’s  veracity  is  quite  apparent. 


6  See  Dabkowski  et  al.  (2014)  for  details  on  the  Girvan-Newman  algorithm. 
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Figure  6.  Permuted  SV3  for  the  Hypothetical  System 

Given  this  partition,  we  return  to  Panel  6  of  Figure  5  and  note  that  U’s  connections 
with  {Q,  L},  {I,  K,  T,  B},  and  {F,  P,  C}  span  the  three  communities,  minimally  adding  five 
intercommunity  interfaces  to  the  existing  architecture  (if  U  is  assigned  to  community  2).  With 
just  seven  intercommunity  interfaces  prior  to  the  addition  of  subsystem  U,  this  seems 
extremely  unlikely.  To  the  contrary,  in  Panel  3  of  Figure  5,  U’s  connections  to  Q,  L,  and  M 
are  consistent  with  its  assignment  to  community  1 ,  and  its  connection  to  B  reinforces  the 
only  intercommunity  interface  between  communities  1  and  2  in  the  existing  architecture 
(interface  J-B).  Unlike  Panel  6  of  Figure  5,  this  seems  plausible. 

In  light  of  this,  community  membership  matters,  and  partitioning  the  architecture  prior 
to  assessing  U’s  degree,  adjacency,  and  weights  is  implied.  Accordingly,  we  elected  to 
apply  the  Girvan-Newman  algorithm  as  a  preprocessing  step,  and  we  subsequently  treat 
intracommunity  and  intercommunity  growth  separately.  Nonetheless,  regardless  of  the  type 
of  growth,  we  can  address  our  previously  identified  growth  modeling  questions  as  follows: 

a.  Degree:  To  model  a  “rich-by-birth”  effect,  view  the  degree  of  U  (Du)  as 
a  random  variable  with  a  probability  mass  function  (pmf)  equal  to  the 
observed  degree  distribution  of  the  existing  system; 

b.  Adjacency:  To  incorporate  a  “rich-get-richer”  effect,  utilize  the 
Barabasi-Albert  preferential  attachment  (PA)  model  from  network 
science,  where  highly  connected  subsystems  are  more  likely  to 
interface  with  U;  and 

c.  Weights:  To  mimic  the  observed  complexity  in  the  existing 
architecture,  cast  the  complexity  of  the  interface  between  U  and 
subsystem  i  (wju)  as  a  conditional  random  variable,  where  the  pmf  for 
Wlu  equates  to  the  observed  interface  complexity  distribution  of 
subsystem  i. 

Taken  together,  our  mechanism  for  generating  unforeseen  architectural  growth  is  as 

follows: 


Preprocessing 

1 .  Initialize  the  system  as  the  current  system, 

2.  Use  Girvan-Newman  to  identify  architectural  communities, 

3.  Randomly  assign  U  to  community  j, 
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Intracommunity  Growth 

4.  Generate  a  realization  for  Duintra,  given  U  is  assigned  to  community  j  (dintra), 

5.  Connect  U  to  dintrasubsystems  inside  community  j  using  the  PA  model, 

6.  For  each  interface  established  in  (5),  assign  complexity  (WjUantra), 

Intercommunity  Growth 

7.  Generate  a  realization  for  Dinter  given  U  is  assigned  to  community  j  (dinter), 

8.  Connect  U  to  dinter  communities  using  the  PA  model,  and 

9.  For  each  interface  established  in  (8),  assign  complexity  (M^u  inter). 

Estimating  the  Cost  of  Architectural  Growth 

The  algorithm  described  in  steps  (1)  through  (9)  generates  a  single  realization  of  the 
architectural  growth  inspired  by  the  addition  of  subsystem  U.  Of  course,  we  ultimately  want 
to  estimate  the  cost  of  these  architectural  changes;  therefore,  we  add  steps  (10)  and  (11),  to 
wit: 

Cost  Estimation 

10.  Estimate  the  cost  for  the  augmented  system  using  COSYSMO  (PMNS*),  and 

1 1 .  Calculate  the  additional  cost  of  adding  subsystem  U  (PMNS*  -  PMNS). 

Furthermore,  as  each  realization  of  cost  is  drawn  from  an  unknown,  underlying 
probability  distribution,  we  add  a  final,  looping  step  ((12)  Store  results  and  return  to  (3)),  and 
we  repeat  the  procedure  a  large  number  of  times.  Implemented  in  the  statistical  software  R, 
the  results  of  10,000  iterations  are  given  in  Figure  7. 


Figure  7.  Empirical  Cumulative  Distribution  Functions  and  Statistics  for  the  Cost  of 

Adding  Subsystem  U  to  Communities  1,  2,  and  3 

As  seen  in  Figure  7,  if  subsystem  U  is  added  to  architectural  community  1 ,  a 
plausible  range  of  values  for  the  expected,  marginal  SE  effort  is  2.68  to  2.78  PMns  (the  95% 
confidence  interval  on  the  mean),  while  the  expected  (or  mean)  SE  cost  is  $54,740. 
Moreover,  as  the  expected  value  of  the  squared  difference  between  a  random  variable’s 
mean  and  itself  is  minimal,  this  value  can  be  interpreted  as  a  “best  guess.”  Alternatively,  if  a 
decision  maker  requires  an  upper  bound  for  the  SE  cost  necessary  to  add  subsystem  U  to 
community  1,  $160,020  (the  largest  observed  simulated  value)  is  a  reasonable  estimate. 
Similar  interpretations  apply  to  Panels  2  and  3  of  Figure  7,  where  the  greater  complexity  of 
communities  2  and  3  generates  higher  costs. 
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Limitations 

While  still  in  its  infancy,  the  above  methodology  has  been  well  received  by  both 
industry  and  academia,  as  it  provides  a  new  tool  for  Pre-MS  A  cost  estimation,  prior  to 
settling  on  a  design.  Nonetheless,  as  with  any  model,  it  is  not  a  panacea,  and  it  has 
numerous  limitations.  For  example,  in  the  hypothetical  system  portrayed  in  Figure  4,  we 
have  assumed  the  existing  architecture’s  interface  complexities  are  known  a  priori  and  with 
certainty.  Unfortunately,  both  are  questionable.  Specifically,  in  its  current  form,  DoDAF  does 
not  require  the  complexity  of  a  system’s  interfaces  to  be  assessed,  and,  even  if  such  an 
assessment  occurred,  uniformly  treating  existing  interface  complexities  as  deterministic 
seems  artificial.  That  said,  neither  of  these  issues  are  insurmountable,  as  interface 
complexity  can  minimally  be  assessed  through  expert  opinion,  and  multiple  expert  opinions 
can  be  stochastically  modeled  via  opinion  pools. 

Additionally,  by  employing  COSYSMO,  the  above  methodology  only  estimates  the 
SE  effort  (and  cost)  required  to  connect  a  new  subsystem;  it  does  not  estimate  total  cost. 
While  this  seemingly  limits  the  model’s  utility,  industry  has  found  SE  effort  to  be  a  valuable 
proxy  for  total  system  cost  (Honour,  2004).  Exploiting  this  relationship,  Lockheed  Martin’s 
Proxy  Estimation  Costing  for  Systems  (PECS)  method  uses  COSYSMO  to  generate  an 
initial  estimate  of  SE  effort,  converts  it  to  program  effort,  and  subsequently  integrates 
additional  cost  parameters  to  estimate  total  system  cost  (Cole,  2012).  A  similar  approach, 
after  appropriate  modification,  would  almost  certainly  work  here. 

Lastly,  in  its  current  form,  the  above  methodology  is  entirely  stochastic,  and  this  may 
inadvertently  cause  unrealistic  connections.  For  instance,  consider  the  SV2  and  SV3  for  the 
OOI  Portland  site  given  in  Figures  2  and  3,  respectively.  By  inspection,  every  subsystem  in 
this  architecture  is  minimally  connected  to  one  of  three  subsystems,  namely:  Core  Switch 
#1,  Control/Operations  Switch,  and  Infrastructure  Management  Switch.  They  are  hubs.  With 
this  in  mind,  if  we  add  a  new  subsystem  to  the  existing  architecture,  it  is  reasonable  (and 
perhaps  mandatory)  to  assume  the  new  subsystem  will  minimally  connect  to  one  of  these 
three  hubs.  While  the  PA  model  in  our  current  methodology  makes  this  likely,  it  is  not 
guaranteed.  As  such,  identifying  design  imperatives  upfront  and  incorporating  these  rules 
into  the  algorithm  can  improve  its  realism  and  accuracy. 

Future  Work 

As  previously  mentioned,  the  current  methodology  utilizes  just  1  of  the  25  DoDAF 
models  required  in  the  CDD.  Thus,  we  are  currently  exploring  the  relationship  between  the 
remaining  24  and  COSYSMO,  with  the  ultimate  goal  of  mapping  relevant  artifacts  to 
COSYSMO’s  CER  parameters.  Similarly,  beyond  the  draft  CDD,  there  are  many  other 
acquisition  documents  available  Pre-MS  A  in  either  approved  (i.e.,  Initial  Capabilities 
Document  (ICD),  Market  Research,  etc.)  or  draft  form  (i.e.,  Analysis  of  Alternatives, 
Consideration  of  Technology  Issues,  Cooperative  Opportunities,  etc.;  Under  Secretary  of 
Defense  [AT&L],  2013b).  Accordingly,  we  will  be  examining  these  documents  for  information 
that  provides  additional,  useful  input  for  Pre-MS  A  cost  estimation.  Finally,  as  with 
Dabkowski  et  al.  (2014),  validating  the  current  methodology  (especially  the  architectural 
growth  mechanism)  remains  a  priority.  As  such,  we  are  using  the  Army’s  recently  launched 
Army  Capability  Architecture  Development  and  Integration  Environment  (ArCADIE) 
database  to  analyze  the  SV2’s  of  unclassified  programs. 

Conclusion 

Given  recent  budgetary  stress  and  increased  program  scrutiny,  the  need  for  better 
cost  estimation,  earlier  in  the  systems  life  cycle,  is  paramount.  Unfortunately,  due  to 
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complications  stemming  from  a  lack  of  system  specification,  Pre-MS  A  cost  estimation 
remains  challenging.  That  said,  the  recently  signed  interim  DoD  Instruction  5000.02  now 
requires  a  DoD  component-approved  CDD  prior  to  MS  A  (Under  Secretary  of  Defense 
[AT&L],  2013b,  p.  46),  and  this  document  must  contain  25  of  the  31  DoDAF  models  ever 
required  by  JCIDS  (CJCS,  2012a,  p.  B-F-6).  Among  these  models  is  the  SV2,  which  visually 
captures  how  the  subsystems  of  a  larger  system  (or  program)  connect.  Compactly 
represented  as  the  SV3,  this  artifact  provides  an  abstract  representation  of  internal  and 
external  system  complexity,  a  factor  identified  as  a  primary  cause  of  cost  overruns  (National 
Research  Council,  2008,  pp.  82-85). 

Accordingly,  in  this  paper  we  posited  and  demonstrated  a  novel  algorithm  for 
estimating  the  cost  of  architectural  growth  (and  more  generally  change)  early  in  the  system 
life  cycle  when  uncertainty  is  high.  Specifically,  drawing  on  the  disparate  disciplines  of 
systems  engineering,  parametric  cost  estimation,  and  network  science,  we  treated  the  SV3 
as  an  adjacency  matrix  and  modeled  the  architectural  change  stemming  from  the  addition  of 
a  subsystem.  Following  this  growth,  we  estimated  the  marginal  increase  in  SE  effort  via  an 
explicit  connection  between  the  open  academic  cost  model  COSYSMO  and  the  SV3. 
Although  our  approach  has  limitations,  its  principal  shortcomings  are  largely  addressable  by 
known  methods.  Nonetheless,  this  work  serves  as  a  proof  of  concept,  and  substantial 
calibration  and  validation  effort  is  required  prior  to  implementation. 
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Agenda 

•  Purpose,  Question,  and  Contribution 

•  Political  and  Acquisition  Background 

•  Peril  of  Pre-MS  A  Cost  Estimation 

•  Systems  Engineering  (SE)  -  A  Discipline  with  Promise 

•  Constructive  Systems  Engineering  Cost  Model  -  A  Tool  for 
Costing  Architectural  Complexity 

•  Network  Science  -  A  Mechanism  for  Generating  Unforeseen 
Architectural  Growth 

•  Simulating  Growth  and  Estimating  Cost 

•  Limitations 

•  Future  work 

•  Questions 

- The  University  of  Arizona.  „ 
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Overarching  Purpose:  To  transform  Model-Based  System 
Engineering  (MBSE)  artifacts  into  computational  knowledge 
that  can  be  leveraged  early  in  the  system  lifecycle  when 
uncertainty  is  high  and  confidence  is  low 

* 

Focused  Question:  Can  parametric  cost  estimation,  in 
conjunction  with  DoD  Architecture  Framework  (DoDAF) 
models,  capture  the  monetary  impact  of  architectural  changes 
early  in  the  system  lifecycle? 

* 

Principal  Contribution:  A  network  science-based  algorithm  for 
estimating  the  cost  of  unforeseen  architectural  growth 

- The  University  of  Arizona,  . 
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Sequence  of  Events 


Situation 


Flirting  with  a  cliff 


Federal  spending 
unsustainable  /  national 
debt  exploding 

Solution 

August  2.  2011:  POTUS 
signs  Budget  Control  Act 
into  law  with  threat  of 
sequestration 

Action  (or  Inaction) 

Congress  fails  to  pass  a 
deficit  reduction  bill 

Reaction 

March  1.  2013:  As 
promised,  POTUS 
implements  cuts 

Impact  on  DoD  Funding 

*  Unanticipated  23%  reduction  in 
Fiscal  Year  2013  budget 

*  Combined  $492  billion  loss  in 
funding  over  next  10  years 

Second  Order  Effects 

*  Furloughs 

*  Reduced  production 

*  Difficult  modernization  decisions  . . . 
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Single  Largest  Impact 

"single  largest  impact  of  sequestration  and  current  budgetary 
unknowns  is  [their  effect  on]  .  .  .  the  meticulous  cost  and 
schedule  planning  mandated  in  numerous  public  laws  and 
DoD  acquisition  policy  directives"* 


Dr.  William  LaPlante 
Principal  Deputy  Assistant  Secretary 
of  the  Air  Force  (Acquisition) 


LTG  Michael  Moeller 
USAF  Deputy  Chief  of  Staff, 
Strategic  Plans  and  Programs 


*  On  impacts  of  a  continuing  resolution  and  sequestration  on  acquisition,  programming,  and  the  industrial  base:  Hearing  before  the 
Subcommittee  on  Tactical  Air  and  Land  Forces,  Committee  on  Armed  Services,  House  of  Representatives,  113th  Cong.,  1st  Sess.,  12 
(2013). 
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We  find  ourselves  in  challenging  times 

•  Even  before  the  cuts,  meticulous  cost  planning  was  tough 

-  Between  1997  and  2009,  47  major  defense  acquisition  programs 
(MDAPs)  experienced  cost  overruns  of  at  least  15%  or  30%  over  their 
current  or  original  baseline  estimates* 

•  Conditions  have  not  improved 

The  Requirement 

In  an  uncertain  environment,  the  need  to  plan  well  is  paramount 


The  Reality 

Budget  uncertain  and 
presumably  shrinking 


Uncertain  funding 
frustrates  cost  planning 


Cost  planning 
already  difficult 


*  Government  Accountability  Office.  (2011).  DOD  COST  OVERRUNS:  Trends  in  Nunn-McCurdy  Breaches  and  Tools  to  Manage  Weapon 
Systems  Acquisition  Costs. 
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Leaning  forward  in  the  foxhole 

•  Despite  recent  challenges,  Congress  recognized  need  for  change  in 
2009  with  the  Weapon  Systems  Acquisition  Reform  Act  (WSARA) 

•  4  organizational  changes  and  7  procedural  adjustments,  including: 

-  Increased  rigor  of  Pre-Milestone  A  (Pre-MS  A)  cost  analysis 

-  Greater  emphasis  on  integration  and  design  risk  by  (1)  establishing 
Performance  Assessments  and  Root  Cause  Analysis  (PARCA)  official 
and  (2)  requiring  the  Director  of  Defense  Research  and  Engineering 
(DDR&E)  to  develop  standards  to  measure  integration  risk 

•  Majority  of  program's  future,  life-cycle  costs  are  often  (perhaps 
unwittingly)  committed  in  preliminary  design  phase 

Today's  design  decisions ,  driven  by  current  requirements ,  determine 
tomorrow's  debts  =>  make  better  design  decisions  now 

The  University  of  Arizona, 


Pre-MS  A  cost  estimation  is  fraught  with  peril 


•  Early  life  cycle  characterized  by  uncertainty* 

-  Requirements  change:  Average  of  28%  of  a  system's  baseline 
requirements  will  change 

-  Changes  often  come  late:  Roughly  43%  of  changes  occur  in  the 
development  phase 

-  More  likely  to  add  than  take  away:  86%  of  changes  manifest  as 
modification  of  an  existing  requirement  or  addition  of  a  new 
requirement 

•  Post  MS-B  requirement  changes  account  for  an  average  of 
21.5%  of  a  program's  total  cost  growth  over  its  MS  B 
estimatet 


*  Pena,  M.,  &  Valerdi,  R.  (2014,  in  press).  Characterizing  the  Impact  of  Requirements  Volatility  on  Systems  Engineering  Effort. 
Systems  Engineering. 

+  Bolten,  J.  G.,  Leonard,  R.  S.,  Arena  M.  V.,  Younossi,  O.,  &  Sollinger,  J.  M.  (2008).  Sources  of  Weapon  System  Cost  Growth  -  Analysis 
of  35  Major  Defense  Acquisition  Programs.  Santa  Monica,  CA:  RAND  Corporation. 


The  University  of  Arizona.  o 
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But  More  Information  is  Available  Earlier 

•  Interim  DoDI  5000.02  now  requires  a  "draft"  or  DoD  component-approved 
Capability  Development  Document  (CDD)  prior  to  MS  A 

•  CDDs  must  contain  25  of  the  31  DoD  Architecture  Framework  (DoDAF) 
models  ever  required  by  JCIDS  . . .  the  same  set  required  by  the  Capability 
Production  Document  (CPD) 


Materiel 
Development 
~K  Decision 


Legend 

=  Decision  Point 

A  =  Milestone  Decision 

|~~*[  =  Requirements  Document 

I  =  Requirements  Authority 
' - '  Review 


•  Or  Equivalent  Approved/Validated  Requirements  Document. 


Engineering  &  Manufacturing 
Development  Phase 

Capability 
Production 
Document* 


A 


What,  if  anything,  can 
these  draft  DoDAF 
diagrams  tell  us  about  cost? 


Production  & 
Deployment  Phase 


Operations  &  Support  | 
Phase 
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What  is  DoDAF  Besides  Mandatory? 

•  DoDAF  is  . . . 

-  A  conceptual  modeling  paradigm  that  provides  common 
understanding* 

-  In  its  third  major  revision  (vl.O  — >  vl.5  v2.0  (v2.02)) 

-  Represented  as  52  models  organized  into  8  collections  (viewpoints) 

-  Designed  to  be  data  versus  product-driven 

-  Meant  to  be  "fit-for-purpose"  versus  rigid 

-  Serving  as  the  DoD's  foundation  for  Model-Based  System  Engineering 

-  Implemented  in  sophisticated  SE  tools  (Atego's  Artisan  Studio,  IBM's 
Rational  Systems  Architect,  Vitech's  CORE,  etc.) 

-  Potentially  overwhelming 

The  University  of  Arizona 
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*  http://dodcio.defense.gov/dodaf20/dodaf20_background.aspx 


DoDAF  v2.02  as  a  Network 


OV-6b 

OV-6a 
OV-5a/5b 
OV-4 
OV-3 


AV-1 


OV-2 
OV-1 


DIV-2 


■2 

SV-3 

SV-4 

SV-5a 

SV-5b 
SV-6 


SV-7 
SV-8 
SV-9 


SvcV-1 
SvcV-lOa 
SvcV-1  Qb 
SvcV'1  Oc 
SvcV-2 
SvcV-3a/3b 
Svc\M 
SvcV-5 


SvcV-9 


SvcV-8 


SvcV-7 


SvcV-6 


Legend 


•  -  models  from  vl  .5  replicated  in  v2.02 

•  -  models  from  vl  .5  renamed  in  v2.02 
O  -  models  from  vl  .5  bifurcated  in  v2.02 
O  -  new  models  in  v2.02 

—  -  explicit  connections  from  vl  .5  manual 

-  -  implicit  connections  from  v2.02  manual 


AV  -  all  (2  models) 

CV  -  capability  (7  models) 

DIV  -  data  and  information  (3  models) 
OV  -  operational  (9  models) 


PV  -  project  (3  models) 
StdV  -  standards  (2  models) 
SvcV  -  services  (13  models) 
SV  -  systems  (13  models) 
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Why  do  we  need  DoDAF  Pre-MS  A? 


•  DoD  procurements  are  often  large,  complex,  and  expensive 


F-35  Joint  Strike  Fighter  (JSF} 

•  3  major  contractors  (LM,  NG,  BAE)* 

•  9  partner  nations* 

•  24  million  lines  of  codeA 


•  105  interfaces1" 

•  Total  lifecycle  cost «  $1 .51 T* 


•  When  changing  complex  systems, 
earlier  is  easier,  but  change 
requires  knowledge5 

DoDAF  Pre-MS  A  helps  close  this  gap 


*  http://www.jsf.mil/f35/ 

A  Hagen,  C.  &  Sorenson,  J.  (2013).  Delivering  Military  Software  Affordably.  Defense  AT&L  Magazine , 
March-April  2013,  30-34. 

t  Becz  et  al.  (2010).  Design  System  for  Managing  Complexity  in  Aerospace  Systems.  Paper  presented  at 
the  2010  AIAA  ATIO/ISSMO  Conference,  Fort  Worth,  Texas. 

4  http://articles.chicagotribune.corn/2012-04-02/news/sns-rt-us-lockheed-fighterbre8310wb- 
20120402_l_problems-or-cost-increases-technical-problems-or-cost-f-35. 

§  Blanchard,  B.,  &  Fabrycky  W.  (1998).  Systems  engineering  and  analysis  (3rd  ed.).  Upper  Saddle  River, 
NJ:  Prentice  Hall. 


Figure  2.11  from  Blanchard  &  Fabrycky  (1998). 


The  University  of  Arizona. 


Systems  Engineering  (SE)  -  A  Discipline  with  Promise 


•  Consider  the  SV-3:  Systems-Systems  Matrix, 
which  compactly  captures  how  the 
subsystems  of  a  larger  system  connect 

•  SV3  falls  within  purview  of  SE 


•  SE  holds  promise  for  controlling  cost 

-  DoD  Cost  Overruns :  "early  and  continued  SE 
analysis"  helps  contain  cost  growth 

-  2008  National  Research  Council  (NRC)  report : 
"application  of  SE  to  decisions  made  in  the  pre- 
Milestone  A  period  is  critical  to  avoiding  (or  at 
least  minimizing)  cost  and  schedule  overruns" 

-  ROI  ofSE:  ROI  for  greater  SE  effort  as  high  as  8:1 


SV3 


-  WSARA  2009 :  Added  a  Director  of  SE  to  the 
USD(AT&L)'s  staff 


The  University  of  Arizona 
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Constructive  Systems  Engineering  Cost  Model  (COSYSMO) 
-  A  Tool  for  Costing  Architectural  Complexity 


Open  academic  parametric  cost  model 

CER  incorporates  size  of  the  system  and  SE  effort  required 


PMns  =  A 


4 

Y  Y™^ 

\i£{e,n,d}k= 1 


E 


size 


14 

n  em> 

j= 1 

effort 


•  SV-3  directly  related  to  the  "number  of  interfaces"  size  driver 

•  For  complexity,  all  interfaces  are  not  created  equal 


Interfaces 

Complexity 

Characteristics 

Easy 

Nominal 

Difficult 

Message  complexity 

Simple 

Moderate 

Complex 

Coupling  level 

Uncoupled 

Loose 

Tight 

Stakeholder  consensus 

Strong 

Moderate 

Low 

Behavior 

Well  behaved 

Predictable 

Emergent 

Relative  weight  (wik) 

1.1 

2.8 

6.3 

The  University  of  Arizona. 
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Hypothetical  SV-3 


•  20  subsystems  with  47  interfaces 
of  varying  complexity 

•  Without  loss  of  generality, 
assume  there  are  . . . 

-  200  easy,  200  nominal,  and  50 
difficult  requirements 

-  5  difficult  critical  algorithms 

•  Using  additional  wjk  and  EMj 
data,*  apply  CER  to  obtain  an 
initial  estimate  of  PMNS 


Mfti-B 


SV-3 

Interface  Complexity 
□  =  Easy,  □=  Nominal,  ■  =  Difficult 
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What  about  inevitable,  unforeseen  change? 


•  245.27  PM  of  SE  effort  are  required,  and,  if  each  PM  costs  $20,000, 
this  equates  to  $4,905,400 

•  But,  this  is  Pre-MS  A;  what  happens  when  requirements  change? 


What  if  we  add  a 
new  subsystem  to 
the  architecture 
without  knowing 

its  purpose? 


The  University  of  Arizona. 
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The  Analytical  Task 


Graphical  Representation 
of  the  SV-3 

Interface  Complexity 
□  =  Easy,  □=  Nominal,  ■=  Difficult 


If  we  add  a  new  subsystem  U, 
how  will  it  connect  to  the 
existing  architecture? 


a)  How  many  subsystems  should 
U  connect  to  (degree)?, 

b)  Given  U  connects  to  d 
subsystems,  which  d 
subsystems  should  it  connect 
to  (adjacency)?,  and 

c)  Given  U  connects  to  a  specific 
set  of  d  subsystems,  what 
should  the  complexity  of  these 
interfaces  be  (weights)? 


What  will  it  cost? 


The  University  of  Arizona. 


Network  Science  -  A  Mechanism  for  Generating 
Unforeseen  Architectural  Growth 

•  Two  fundamental  assumptions 


1.  Manner  in  which  SEs  architect  complex  systems  matters 

-  Partition:  Search  for  "architectural  communities"  using  Girvan-Newman 

2.  Existing  architecture  foretells  future  architecture 

-  Degree:  Treat  degree  of  U  (DJ  as  a  random  variable  with  a  probability  mass 
function  (pmf)  equal  to  observed  degree  distribution  of  existing  system 
("rich-by-birth"  effect) 

-  Adjacency:  Utilize  Barabasi-Albert  preferential  attachment  (PA)  model, 
where  highly  connected  subsystems  are  more  likely  to  interface  with  U 
("rich-get-richer"  effect);  and 

-  Weights:  Model  complexity  of  interface  between  U  and  subsystem  /  (wiyj)  as 
a  conditional  random  variable,  where  pmf  for  wi[}  equates  to  observed 
interface  complexity  distribution  of  subsystem  / 

The  University  of  Arizona 
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Identifying  Architectural  Communities 

•  Back  to  our  hypothetical  SV3  . . . 

•  3  communities  detected  using  Girvan-Newman 


The  University  of  Arizona. 


a  zeng  meeting - 

Isomorphic  SV3 

•  Permuted  SV3  reinforces  the  partitioning 

OOu/j5-rZOiG^i:i-a)0ii.Q,<a:o 

Q 
O 

E 
L 
M 
J 
N 
G 
H 
D 
I 

K 
T 
B 
S 
F 
P 
A 
R 
C 

Adjacency  Matrix  Representation  Graphical  Representation 

SV3  in  DoDAF 

Interface  Complexity  Interface  Complexity 

□  =  Easy,  □=  Nominal,  ■  =  Difficult  □  =  Easy,  □=  Nominal,  ■  =  Difficult 

- The  University  of  Arizona,. 
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Simulating  Growth  and  Estimating  Cost 


•  Pseudo-code  to  estimate  cost  impact  of  adding  U 


Intracommunity 

growth 


Intercommunity 

growth 


For  a  specified  number  of  iterations  . . . 

1.  Initialize  the  system  as  the  current  system 

2.  Use  Girvan-Newman  to  identify  architectural  communities 

3.  Randomly  assign  U  to  community  j 

'  4.  Generate  a  realization  for  Dv{Jjntra  |  assigned  to  community  j  (djntra) 

■  5.  Connect  U  to  djntra  subsystems  inside  community  j  using  the  PA  model 
_  6.  For  each  interface  established  in  (5),  assign  complexity  (wj[J  jntra) 

7.  Generate  a  realization  for  Dv[Jjnter  |  assigned  to  community  j  (djnter) 

8.  Connect  U  to  djnter  communities  using  the  PA  model 

_  9.  For  each  interface  established  in  (8),  assign  complexity  (wj[J  jnter) 

10.  Estimate  cost  for  augmented  system  using  COSYSMO  (PMNS*) 

11.  Calculate  additional  cost  of  adding  subsystem  U  (PMNS*  -  PMNS) 

12.  Store  results  and  return  to  (3) 


The  University  of  Arizona.  „ 
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Cost  Insights  from  Monte  Carlo  Simulation 

Adding  New  Subsystem  to  Community  1 


OOuij5TZOic„^i-ai(Ou,(i<Q:o 


•  95%  Cl  is  (2.68,  2.78)  in  PMNS 

•  "Best  guess"  for  the  cost  of  adding 
subsystem  U  is  $54,740 

•  Cost  to  add  subsystem  U  should  not  exceed 
$160,020 


New  Subsystem  in  Community  1 


Estimated  Cost  of  Adding  Subsystem  (PMNS ) 


ECDF  for  the  additional  cost  of  adding 
subsystem  U  to  Community  1 


The  University  of  Arizona. 
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Cost  Insights  from  Monte  Carlo  Simulation 

Adding  New  Subsystem  to  Community  2 


zOxo„n:t-co</;u.  cl  <  cc  o 


New  Subsystem  in  Community  2 


95%  Cl  is  (6.25,  6.39)  in  PMNS 

"Best  guess"  for  the  cost  of  adding 
subsystem  U  is  $126,520 

Cost  to  add  subsystem  U  should  not  exceed 
$280,240 


Estimated  Cost  of  Adding  Subsystem  (PMNS  ) 

ECDF  for  the  additional  cost  of  adding 
subsystem  U  to  Community  2 


The  University  of  Arizona 


23 


Cost  Insights  from  Monte  Carlo  Simulation 

Adding  New  Subsystem  to  Community  3 


OOuij5TZOic«^i-fl[iwiLii<(ro 


•  95%  Cl  is  (3.86,  4.00)  in  PMNS 

•  "Best  guess"  for  the  cost  of  adding 
subsystem  U  is  $78,720 

•  Cost  to  add  subsystem  U  should  not  exceed 
$155,120 


New  Subsystem  in  Community  3 


Estimated  Cost  of  Adding  Subsystem  (PMNS ) 

ECDF  for  the  additional  cost  of  adding 
subsystem  U  to  Community  3 


The  University  of  Arizona. 
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A  Few  Limitations 

•  Technological  interfaces  are  not  random;  they  are 
engineered  based  on  requirements 

-  Concur;  however,  detailed  interfaces  are  engineered  AFTER 
requirements  mature;  this  is  an  early  lifecycle  estimate 

•  Addition  of  subsystem  U  could  reasonably  necessitate  the 
"rewiring"  of  the  existing  architecture 

-  Concur;  should  probably  be  treated  as  a  higher  order  effect 

•  Using  existing  architecture  to  estimate  future  architecture 
fails  to  account  for  revolutionary  change 

-  Concur;  however,  our  approach  addresses  evolutionary 
change;  wholesale  redesign  is  outside  the  scope  of  our  work 

- The  University  of  Arizona. 
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Future  Work 

•  Our  immediate  research  efforts  are  . .  . 

1.  Conceptual  Modeling 

-  Adding  more  than  1  subsystem 

-  Rewiring  existing  architecture 

-  Accounting  for  "nodal  properties" 

2.  Exploring  the  relationship  between  COSYSMO  and  the 
remaining  24  DoDAF  models  required  in  the  CDD 

3.  Validation 

-  Does  our  methodology  adequately  model  how 
technical  systems  "grow" 


The  University  of  Arizona, _ 
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Questions 

CORRESPONDING  AUTHOR: 

LTC  Matt  Dabkowski 

mfdl@email.arizona.edu 
matthew.dabkowski(5)us.  armv.mil 

TOPIC  TITLE:  The  Budding  SV3  -  Estimating  the  Cost 
of  Architectural  Growth  Early  in  the  Life  Cycle 

The  University  of  Arizona. 
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Backup  Slides 
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Why  Communities  Matter  -  A  Thought  Experiment 


0B 

oG 

A 

Vc  F~/ 

Start  with  this  system  . . . 

f—  \ . -  -  O - 0 

f  \  ...O 

ECK 

\  / 

—  o°  JG> 

O' 

Run  without  Girvan-Newman  . . . 

,oK 

0G 

And  get  this  .. .  A° 

/  V 

/  \  / 

x\/ 

Eo 

o°  J© 

Existing  Architecture 

New  Architecture 

•2  communities 

•2  communities? 

•Bridging  ties  rare 

•Bridging  ties  rare? 

•Bridging  ties  difficult 

•Bridging  ties  difficult? 

When  it  comes  to  incremental  growth ,  existing  community  structure  matters! 

The  University  of  Arizona 
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Girvan-Newman  Algorithm 

•  Based  on  the  idea  of  edge  betweenness  (eb),  the  number  of 
geodesic  (shortest)  paths  that  contain  an  edge 

•  Edge's  with  high  eb  bridge  communities  of  vertices,  groups  of 
vertices  where  the  number  of  edges  is  dense  within  and  sparse 
between  groups 

•  Sketch  of  Girvan-Newman  (GN) 

1.  Given  network  G  of  size  n,  calculate  eb  for  each  edge  in  G 

2.  Delete  edge  with  the  highest  eb  from  G,  yielding  subgraph  G' 

3.  If  G'  has  0  edges,  terminate;  otherwise,  set  G  as  G'  and  return  to  1 

•  At  termination,  GN  produces  a  dendogram  (tree)  with  n  leaves 

•  Cutting  the  tree  at  different  levels  produces  different  community 
structures 

•  Community  structure  that  maximizes  modularity  is  selected 
- The  University  of  Arizona. 
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A  Simple  Example  (1  of  4) 

Full  Graph  and  Edge  Betweenness 


|  6.83 1  0  D  2.50 


3.50 


B 


2.33  2.83 


1.83 


2.50 


3.33 


4.17 


C  5  17  ^ 

• - 2j-L£ - # 


o 


Full  graph  (G)  has  7  vertices  and  10  edges  Edge  betweenness  (eb)  is  calculated  for  each  edge  in  G 

•  Example  edge  betweenness  calculation  (ebB_c  =  2.5) 

-  B-C  is  the  shortest  path  between  B  and  C  ( ebB_c  =  1) 

-  B-C  is  on  the  unique  shortest  path  between  B  and  F  (B-C-F  ->  ebB_c=  2) 

-  B-C  is  on  1  of  2  shortest  paths  between  B  and  E  (B-A-D-E  and  B-C-F-E  ->  ebB_c  =  2.5) 

•  Edge  A-D  has  the  highest  edge  betweenness  (ebA.D  =  6.83);  delete  A-D  from  G 

The  University  of  Arizona. 
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A  Simple  Example  (2  of  4) 

Edge  Deletion  and  Fragmentation 


•  Edge  A-D  is  deleted  from  G  leaving  G' 

•  G'  has  at  least  one  edge;  G  =  G' 

•  Edge  betweenness  is  calculated  for  G 

•  Edge  C-F  has  the  highest  edge  betweeness 


•  Edge  C-F  is  deleted  from  G  leaving  G' 

•  G'  has  at  least  one  edge;  G  =  G' 

•  G  is  now  fragmented  into  2  communities 


•  After  the  deletion  of  edge  C-F,  G  has  two  communities  (A,B,C)  and  (D,E,F,G) 

•  How  "good"  is  this  division?  Modularity  provides  an  answer . . . 

The  University  of  Arizona 
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A  Simple  Example  (3  of  4) 


q  =  22  (e“  -  a‘ ) 
1=1 


A 

o 


B 


C 

o 


Modularity  (Q) 


D  E 

A 

a 

m 

#— 

o  o 

B 

B 

F  G 

\|  C 

F  G 

0  0 

o  o 

After  the  removal  of  edge  C-F, 

G  is  fragmented  into  2  ( n )  communities 


Using  the  full  graph,  calculate  ejjf  fraction  of 
edges  between  communities  /  and  j  (for  all  i,  j ) 


l  2 


0.3 

0.1 

0.1 

0.5 

Using  e(y,  build  matrix  e 


Row  Sums 


2 

2 

3| 

■f 

®ii "  ai 

0.4 

0.16 

0.14 

0.6 

0.36 

0.14 

Q 

0.28 

Calculate  modularity  (Q),  where  o(  is  the  sum  of  row  /  of  e 
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A  Simple  Example  (4  of  4) 

Termination,  Modularity  Maximization,  and  Community  Selection 

Number  of  Communities 

^“’r-CNCNCOCOTTVOmCDr^- 


Edge  Removed  Den  dog  ram 


•  Process  continues  in  a  similar  manner  until  G  has  no  edges  remaining 

•  At  termination,  the  dendogram  has  7  (n)  leaves  with  6  potential  community  structures 

•  Our  initial  fragmentation  maximizes  modularity  at  0.280 

•  Conclusion:  G  contains  2  communities,  namely,  (A,B,C)  and  (D,E,F,G) 

The  University  of  Arizona, 


Modularity 


